Ai platform with automatic analysis data and methods for use therewith

ABSTRACT

A system operates by: generating, via a machine that includes at least one processor and a non-transitory machine-readable storage medium and utilizing a graphical user interface, a content analysis control panel; receiving, via the machine, content data; detecting, via one or more AI models implemented via the machine, detection data that includes first portions of the content data associated with a protected attribute and second portions of the content data associated with a predetermined metric; generating, via the machine, analysis data associated with the protected attribute and the predetermined metric; and facilitating display, via the content analysis control panel, the analysis data associated with the protected attribute and the predetermined metric.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present U.S. Utility Pat. Application claims priority pursuant to 35U.S.C. § 119(e) to U.S. Provisional Application No. 63/262,395, entitled“AI PLATFORM WITH CUSTOMIZABLE CONTENT ANALYSIS CONTROL PANEL ANDMETHODS FOR USE THEREWITH”, filed Oct. 12, 2021; U.S. ProvisionalApplication No. 63/262,396, entitled “AI PLATFORM WITH CUSTOMIZABLEVIRTUE SCORING MODELS AND METHODS FOR USE THEREWITH”, filed Oct. 12,2021; and U.S. Provisional Application No. 63/262,397, entitled “AIPLATFORM WITH AUTOMATIC ANALYSIS DATA AND METHODS FOR USE THEREWITH”,filed Oct. 12, 2021, all of which are hereby incorporated herein byreference in their entirety and made part of the present U.S. UtilityPat. Application for all purposes.

TECHNICAL FIELD

The present disclosure relates to processing systems and applicationsused in the development, analysis and/or use of artificial intelligencemodels or other content.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1A presents a block diagram representation of an example system.

FIG. 1B presents a block diagram representation of an example artificialintelligence (AI) development platform.

FIG. 1C presents a block diagram representation of an example system.

FIG. 1D presents a block diagram representation of an example contentanalysis platform.

FIG. 1E presents a block diagram representation of an example clientdevice.

FIG. 2A presents a flowchart representation of an example method.

FIG. 2B presents a flowchart representation of an example method.

FIG. 2C presents a flowchart representation of an example method.

FIG. 2D presents a flowchart representation of an example method.

FIG. 3A presents a block diagram representation of an example AI autodetection model.

FIG. 3B presents a block diagram representation of an exampleauto-mapping function.

FIG. 3C presents a block diagram representation of an example virtuescoring model.

FIG. 3D presents a block diagram representation of an example surveycreation widget.

FIG. 3E presents a block diagram representation of an example of controlpanel generation tools.

FIG. 3F presents a pictorial representation of an example of a contentanalysis control panel.

FIGS. 4A - 4Y present graphical diagram representations of examplescreen displays or portions thereof.

FIGS. 5A - 5D present graphical diagram representations of examplescreen displays or portions thereof.

FIGS. 6A - 6F present graphical diagram representations of examplescreen displays or portions thereof.

DETAILED DESCRIPTION

FIG. 1A presents a block diagram representation of an example system inaccordance with various embodiments. In particular, a system 850 ispresented that includes an AI development platform 800 that communicateswith client devices 825 via a network 115. The network 115 can be theInternet or other wide area or local area network, either public orprivate. The client devices 825 can be computing devices of users suchas AI developers or administrators of databases, social media platformsor other sources of AI or media content.

As AI development accelerates at an unprecedented rate, many machinelearning (ML) Engineers are beginning to require knowledge in a diverserange of fields including AI ethics, MLOps, and AutoML. Currently thereis just scattered, disparate toolkits, which can lead developers to makepoor decisions due to lack of experience and accountability.

There is also increased regulation requirements under way by places likethe EU and potential need to meet some standards of quality in the nearfuture. IBM has Fairness 360 for bias. IBM also has the ExplainabilityToolkit for increasing transparency. There is Audit-AI for statisticalbias detection. Lime has software for visualizing bias to increasefairness. There is SHAP that uses game theory for explain output ofblack box model. There is XAI for dynamic systems. The problem is thatmost AI developers do not want to switch from one platform or toolkit,to another, and another again. The AI development platform 800 andsystem 850 makes these technological improvements to computer technologyby reworking the AI infrastructure from the ground up, building AIethics into the work experience, and streamlining the process to achievesafe and effective algorithms for ML developers. It provides a “one stopshop” to building robust and certifiable AI systems. Although theprimary goal of the AI development platform 800 is to provide a softwareas a service (SaaS) platform to an ethical AI community, it may be usedin conjunction with social media platforms such as Instagram, Facebook,LinkedIn, GitHub, etc. This platform could also be used by AI ethiciststo audit their own systems of AI development. Users can use theframework and publicly post their decisions along the way for human inthe loop feedback from a community through the posting of problems,questions, reviews, etc. Furthermore, the systems described hereinimprove computer technology by providing a user interface with many newfeatures and combinations that improve the user experience, increaseuser efficiency and generate more accurate, more robust and morevirtuous results.

The AI development platform 800 includes:

-   a. a platform access subsystem 813 that provides secure access to    the AI development platform to a plurality of client devices 825 via    the network 115;-   b. a learning and collaboration subsystem 811 that provides a    network-based forum that facilitates a collaborative development of    machine learning models or other AI tools via the plurality of    client devices 825 and that, for example, provides access to a    library of AI tutorials, a database of AI news, a forum for    questions and answers regarding machine learning, including the use    of specific machine learning techniques and/or whether or not    particular process is fair, biased, transparent, secure, safe, etc.,    and/or a database of documentation regarding the AI development    platform 800 including, for example, instructions on what the    platform is, why it is, what is in it, who it is for, when to use    it, and how to use it and further including instructions on the use    of the various and subsystems, and/or how to access and operate the    various customizations, interconnected tools/widgets and other    features via the AI development platform 800;-   c. a subscription and billing subsystem 815 that controls access to    the AI development platform 800 via each of the plurality of client    devices 825 in conjunction with subscription information associated    with each of the plurality of client devices 825 and further, that    generates billing information associated with each of the plurality    of client devices 825 in accordance with the subscription    information; and-   d. a privacy management system 817 that protects the privacy of    machine learning development data associated with each of the    plurality of client devices 825.

In operation, the AI development platform 800 facilitates thedevelopment of a training dataset associated with at least one of theplurality of client devices 825 via dataset development tools 802. Theresulting dataset can be stored, for example, in a database 819associated with the AI development platform 800. The AI developmentplatform 800 also provides access to a plurality of auto machinelearning tools 804, such as DataRobot, H20.ai and/or other auto machinelearning tools to facilitate the development of an AI model. The AIdevelopment platform 800 includes a set of control panel generationtools 806 that facilitate the generation and user-customization of agraphical user interface (GUI) based content analysis control panel.

The AI development platform 800 also includes a plurality of AI analysistools/widgets 808 that implement, for example, auto detection andmapping tools such as AI models, statistical functions or other AI orfunctions that analyze input datasets to automatically identify and/ormap data associated with protected attributes, key performanceindicators and/or other metrics. In addition, the AI analysistools/widgets 808 can also include a plurality of standard virtuescoring models that each generate a corresponding virtue score. Examplesof such standard virtue scoring models include a responsibility model,an equitability (or bias) model, a reliability model, an explainabilitymodel, a robustness model, a traceability model and/or other models thatgenerate virtue scores such as a responsibility score, an equitability(or bias) score, a reliability score, an explainability score, and/orother morality or virtue score. In addition, the AI analysistools/widgets 808 can include tools to facilitate the generation of oneor more virtue scoring models, such as ML or other AI models that aregenerated based on survey data and the collection of correspondingsurvey results. Furthermore, the AI analysis tools/widgets 808 caninclude survey widgets and other tools to facilitate the generation ofuser-customized virtue scoring models that can differ from each of thestandard virtue scoring models, and that are implemented via ML or otherAI models that are generated based on user-customized survey data andthe collection of corresponding survey results.

The AI development platform 800 also provides access to a versioncontrol repository 812, such as a Git repository or other versioncontrol system for storing and managing a plurality of versions of thetraining dataset and the AI model. The AI development platform 800 alsoprovides access to one or more machine learning management tools 810 toperform other management operations associated with the AI model,training dataset, etc.

In operation, the content analysis control panel generated via the setof control panel generation tools 806 operates in conjunction with theAI analysis tools/widgets 808 to provide a graphical user interface thataids the user by gathering and presenting AI data and/or other contentfor analysis, the creation of custom virtue scoring models, theselection of particular virtue scoring models (either custom or preset)to be used, and the presentation of virtue scores and other analysisresults. For example, the content analysis control panel operates viathe control panel generation tools 806 and associated AI analysistools/widgets 808 to:

-   guide the user through customization of control panel settings and    customization parameters used to generate the content analysis    control panel;-   facilitate the selection of data sets from an AI model or content    source in addition to the selection of protected attributes, key    performance indicators and/or other metrics;-   identify, map and present data associated with the protected    attributes, key performance indicators and/or other metrics    including a customized selection of statistics, charts, graphs    and/or other visualizations;-   facilitate the generation of survey data and collection of survey    results data to facilitate the generation of custom and/or standard    virtue scoring models;-   generate and present virtue scores associated with a selected group    of customized and/or standard virtue scoring models including a    customized selection of statistics, charts, graphs and/or other    visualizations of each score; and-   generate and present suggested improvements to any of the virtue    scores associated with any of a selected group of virtue scoring    models.

In an example of operation, the AI development platform 800 operates toperform operations that include:

-   generating, via a machine that includes at least one processor and a    non-transitory machine-readable storage medium and utilizing a    graphical user interface, a content analysis control panel;-   receiving, via the machine, customization data that indicates a    plurality of virtue scoring models, and presentation parameters    associated with the plurality of scoring models;-   receiving, via the machine, content data from an AI model or media    source;-   generating, via the machine, predicted virtue score data associated    with the content data for each of the plurality of virtue scoring    models; and-   facilitating display, via the content analysis control panel and in    accordance with settings and other customization data, the predicted    virtue score data associated with the content data for each of the    plurality of virtue scoring models.

In another example of operation, the AI development platform 800operates to perform operations that include:

-   generating, via a machine that includes at least one processor and a    non-transitory machine-readable storage medium and utilizing a    graphical user interface, custom survey data in response to user    interactions with the graphical user interface;-   receiving, via the machine and responsive to the custom survey data,    survey results data;-   generating, utilizing machine learning and via the machine, a    customized virtue scoring model based on the custom survey data and    the survey results data;-   receiving, via the machine, content data from an AI model or media    source;-   generating, via the machine and utilizing the customized virtue    scoring model, predicted virtue score data associated with the    content data; and-   facilitating display, via the graphical user interface, the    predicted virtue score data associated with the content data.

In a further example of operation, the AI development platform 800operates to perform operations that include:

-   generating, via a machine that includes at least one processor and a    non-transitory machine-readable storage medium and utilizing a    graphical user interface, a content analysis control panel;-   receiving, via the machine, content data from an AI model or media    source;-   detecting, via one or more AI models implemented via the machine,    detection data that includes first portions of the content data    associated with a protected attribute and second portions of the    content data associated with a predetermined metric;-   generating, via the machine, analysis data associated with the    protected attribute and the predetermined metric; and-   facilitating display, via the content analysis control panel, the    analysis data associated with the protected attribute and the    predetermined metric.

It should be noted that while the learning and collaboration subsystem811, the platform access subsystem 813, subscription and billingsubsystem 815, the privacy management system 817 and the database 819,the dataset development tools 802, AutoML, tools 804, control panelgeneration tools 806, AI analysis tools/widgets 808, ML management tools810 and the version control repository 812 are shown as being internalto the AI development platform 800, in other examples, any subset of thevarious elements of AI development platform 800 can be implementedexternal to the AI development platform 800 and coupled to the othercomponents via the network 115. Furthermore, the AI development platform800 can be implemented in a cloud computing configuration with any orall of the various elements of AI development platform 800 implementedwithin the cloud.

FIG. 1B presents a block diagram representation of an AI developmentplatform 800 in accordance with various embodiments. In particular, theAI development platform 800 includes a network interface 820 such as a3G, 4G, 5G or other cellular wireless transceiver, a Bluetoothtransceiver, a WiFi transceiver, UltraWideBand transceiver, WIMAXtransceiver, ZigBee transceiver or other wireless interface, a UniversalSerial Bus (USB) interface, an IEEE 1394 Firewire interface, an Ethernetinterface or other wired interface and/or other network card or modemfor communicating for communicating via the network 115.

The AI development platform 800 also includes a processing module 830and memory module 840 that stores an operating system (O/S) 844 such asan Apple, Unix, Linux or Microsoft operating system or other operatingsystem, the learning and collaboration subsystem 811, the platformaccess subsystem 813, subscription and billing subsystem 815, theprivacy management system 817 and the database 819, the datasetdevelopment tools 802, AutoML, tools 804, control panel generation tools806, AI analysis tools/widgets 808, ML management tools 810 and theversion control repository 812. In particular, the O/S 844, the learningand collaboration subsystem 811, the platform access subsystem 813,subscription and billing subsystem 815, the privacy management system817 and the database 819, the dataset development tools 802, AutoML,tools 804, control panel generation tools 806, AI analysis tools/widgets808, ML management tools 810 and the version control repository 812 eachinclude operational instructions that, when executed by the processingmodule 830, cooperate to configure the processing module 830 into aspecial purpose device to perform the particular functions of the AIdevelopment platform 800 described herein.

The AI development platform 800 may include a user interface (I/F) 862such as a display device, touch screen, key pad, touch pad, joy stick,thumb wheel, a mouse, one or more buttons, a speaker, a microphone, anaccelerometer, gyroscope or other motion or position sensor, videocamera or other interface devices that provide information to a user ofthe AI development platform 800 and that generate data in response tothe user’s interaction with AI development platform 800.

The processing module 830 can be implemented via a single processingdevice or a plurality of processing devices. Such processing devices caninclude a microprocessor, microcontroller, digital signal processor,microcomputer, central processing unit, quantum computing device, fieldprogrammable gate array, programmable logic device, state machine, logiccircuitry, analog circuitry, digital circuitry, and/or any device thatmanipulates signals (analog and/or digital) based on operationalinstructions that are stored in a memory, such as memory 840. The memorymodule 840 can include a hard disc drive or other disc drive, read-onlymemory, random access memory, volatile memory, non-volatile memory,static memory, dynamic memory, flash memory, cache memory, and/or anydevice that stores digital information. Note that when the processingdevice implements one or more of its functions via a state machine,analog circuitry, digital circuitry, and/or logic circuitry, the memorystoring the corresponding operational instructions may be embeddedwithin, or external to, the circuitry comprising the state machine,analog circuitry, digital circuitry, and/or logic circuitry. While aparticular bus architecture is presented that includes a single bus 860,other architectures are possible including additional data buses and/ordirect connectivity between one or more elements. Further, the AIdevelopment platform 800 can include one or more additional elementsthat are not specifically shown.

FIG. 1C presents a block diagram representation of an example system. Inparticular a content analysis system 865 is shown that includes severalelements of the AI development platform 800 that are referred to bycommon reference numerals. Similarly, FIG. 1D presents a block diagramrepresentation of an example content analysis platform 875 is shown thatincludes several elements of the AI development platform 800 that arereferred to by common reference numerals.

While the discussions of the AI development platform 800 have focused onthe development and analysis of AI models, it should be noted than manyof the elements of the AI development platform 800 also apply to theanalysis of other media content that may or may not be AI related. Thecontent analysis system 865, for example, includes content analysistools/widgets 808′ that includes the same or similar tools to the AIanalysis tools/widgets 808, but that operate on media content or othercontent data, be it AI generated or not.

FIG. 1E presents a block diagram representation of an example clientdevice in accordance with various embodiments. In particular, a clientdevice 825 is presented that includes a network interface 220 such as a3G, 4G, 5G or other cellular wireless transceiver, a Bluetoothtransceiver, a WiFi transceiver, UltraWideBand transceiver, WIMAXtransceiver, ZigBee transceiver or other wireless interface, a UniversalSerial Bus (USB) interface, an IEEE 1394 Firewire interface, an Ethernetinterface or other wired interface and/or other network card or modemfor communicating for communicating via network 115.

The client device 825 also includes a processing module 230 and memorymodule 240 that stores an operating system (O/S) 244 such as an Apple,Unix, Linux or Microsoft operating system or other operating system,training data 120, and one or more gaming applications 248. Inparticular, the O/S 244 and gaming application 248 each includeoperational instructions that, when executed by the processing module230, cooperate to configure the processing module into a special purposedevice to perform the particular functions of the client device 825described herein.

The client device 825 also includes a user interface (I/F) 262 such as adisplay device, touch screen, key pad, touch pad, joy stick, thumbwheel, a mouse, one or more buttons, a speaker, a microphone, anaccelerometer, gyroscope or other motion or position sensor, videocamera or other interface devices that provide information to a user ofthe client device 825 and that generate data in response to the user’sinteraction with the client device 825.

The processing module 230 can be implemented via a single processingdevice or a plurality of processing devices. Such processing devices caninclude a microprocessor, microcontroller, digital signal processor,microcomputer, central processing unit, quantum computing device, fieldprogrammable gate array, programmable logic device, state machine, logiccircuitry, analog circuitry, digital circuitry, and/or any device thatmanipulates signals (analog and/or digital) based on operationalinstructions that are stored in a memory, such as memory 240. The memorymodule 240 can include a hard disc drive or other disc drive, read-onlymemory, random access memory, volatile memory, non-volatile memory,static memory, dynamic memory, flash memory, cache memory, and/or anydevice that stores digital information. Note that when the processingdevice implements one or more of its functions via a state machine,analog circuitry, digital circuitry, and/or logic circuitry, the memorystoring the corresponding operational instructions may be embeddedwithin, or external to, the circuitry comprising the state machine,analog circuitry, digital circuitry, and/or logic circuitry. While aparticular bus architecture is presented that includes a single bus 260,other architectures are possible including additional data buses and/ordirect connectivity between one or more elements. Further, the clientdevice 825 can include one or more additional elements that are notspecifically shown.

The client device 825 operates, via network interface 220, network 115and AI development platform 800 and/or content analysis platform 875. Invarious embodiments, the client device 825 operates to display agraphical user interface, such as a content analysis control panel orother user interface. For example, the client device 825 displays acontent analysis control panel based on content analysis control paneldata generated by either the AI analysis platform 800 or the contentanalysis platform 875 and, in particular, the graphical user interfacecan display one or more screen displays based on data generated by theAI development platform 800 and/or content analysis platform 875.Furthermore, the graphical user interface can operate in response tointeractions by a user to generate input data that is sent to the AIdevelopment platform 800 and/or content analysis platform 875 to controlthe operation of the AI development platform 800 and/or content analysisplatform 875 and/or to provide other input.

FIG. 2A presents a flowchart representation of an example method inaccordance with various embodiments. In particular, a method 600 ispresented for use with any of the functions and features discussed inconjunction with FIGS. 1A-1E. Furthermore, a system comprising a networkinterface configured to communicate via a network, at least oneprocessor and a non-transitory machine-readable storage medium can storeoperational instructions that, when executed by the at least oneprocessor, cause the at least one processor to perform operations thatany of the method steps described below.

Step 602 includes providing, via a system that includes a processor anda network interface, an AI development platform that includes: aplatform access subsystem that provides secure access to the AIdevelopment platform to a plurality of client devices via the networkinterface; a learning and collaboration subsystem that provides anetwork-based forum that facilitates a collaborative development ofmachine learning tools via the plurality of client devices and thatprovides access to a library of AI tutorials and a database of AI news;a subscription and billing subsystem that controls access to the AIdevelopment platform via each of the plurality of client devices inconjunction with subscription information associated with each of theplurality of client devices and further; that generates billinginformation associated with each of the plurality of client devices inaccordance with the subscription information; and a privacy managementsystem that protects the privacy of machine learning development dataassociated with each of the plurality of client devices.

Step 604 includes facilitating, via the AI development platform, thedevelopment of a training dataset associated with at least one of theplurality of client devices. Step 606 includes providing, via the AIdevelopment platform, access to a plurality of auto machine learningtools to facilitate the development of an AI model. Step 608 includesproviding, via the AI development platform, access to a plurality of AIanalysis widgets to facilitate the evaluation of the AI model, whereinthe plurality of AI analysis widgets include a plurality of virtuescoring models that predict virtue scores for the AI model associatedwith the plurality of virtues. Step 610 includes providing, via the AIdevelopment platform, access to a version control repository for storinga plurality of versions of the training dataset and the AI model.

FIG. 2B presents a flowchart representation of an example method inaccordance with various embodiments. In particular, a method 620 ispresented for use with any of the functions and features discussed inconjunction with FIGS. 1A-1E and/or the method of FIG. 2A. Furthermore,a system comprising a network interface configured to communicate via anetwork, at least one processor and a non-transitory machine-readablestorage medium can store operational instructions that, when executed bythe at least one processor, cause the at least one processor to performoperations that any of the method steps described below.

Step 622 includes generating, via a machine that includes at least oneprocessor and a non-transitory machine-readable storage medium andutilizing a graphical user interface, a content analysis control panel.Step 624 includes receiving, via the machine, customization data thatindicates a plurality of virtue scoring models, and presentationparameters associated with the plurality of scoring models.

Step 626 includes receiving, via the machine, content data. Step 628includes generating, via the machine, predicted virtue score dataassociated with the content data for each of the plurality of virtuescoring models. Step 630 includes facilitating display, via the contentanalysis control panel and in accordance with the customization data,the predicted virtue score data associated with the content data foreach of the plurality of virtue scoring models.

In addition or in the alternative, the plurality of virtue scoringmodels include a plurality of artificial intelligence (AI) models thatare each trained based on survey data to generate portions of thepredicted virtue score data indicating a corresponding one of aplurality of scores.

In addition or in the alternative, the plurality of AI models includes aresponsibility model and the plurality of scores includes aresponsibility score that is based on an amount the content dataaddresses legal or ethical principles.

In addition or in the alternative, the plurality of AI models includesan equitability model and the plurality of scores includes anequitability score that is based on an amount of bias in the contentdata.

In addition or in the alternative, the plurality of AI models includes areliability model and the plurality of scores includes a reliabilityscore that indicates variations in others of the plurality of scores.

In addition or in the alternative, the plurality of AI models includesan explainability model and the plurality of scores includes anexplainability score associated with the content data.

In addition or in the alternative, the plurality of AI models includes amorality model and the plurality of scores includes a morality scoreassociated with the content data.

In addition or in the alternative, the method can further includegenerating improvement data associated with at least one of theplurality of scores.

In addition or in the alternative, the content data is an ArtificialIntelligence (AI) model.

In addition or in the alternative, the presentation parameters includesa customized selection of at least one of: at least one statistic, atleast one chart, or at least one graph.

In addition or in the alternative, the method can further includedisplaying, via the content analysis control panel and in accordancewith the customization data, of at least one of: at least one protectedattribute, or at least one key performance indicator.

In addition or in the alternative, the method can further includefacilitating selection of the content data from at least one of: an AImodel, or a content source.

In addition or in the alternative, the method can further includegenerating, based on user input, survey data corresponding to a survey;collecting survey results data in response to the survey; andfacilitating generation of a custom virtue scoring model of theplurality of virtue scoring models.

FIG. 2C presents a flowchart representation of an example method inaccordance with various embodiments. In particular, a method 640 ispresented for use with any of the functions and features discussed inconjunction with FIGS. 1A-1E and/or the methods of FIGS. 2A and/or 2B.Furthermore, a system comprising a network interface configured tocommunicate via a network, at least one processor and a non-transitorymachine-readable storage medium can store operational instructions that,when executed by the at least one processor, cause the at least oneprocessor to perform operations that any of the method steps describedbelow.

Step 642 includes generating, via a machine that includes at least oneprocessor and a non-transitory machine-readable storage medium andutilizing a graphical user interface, custom survey data in response touser interactions with the graphical user interface. Step 644 includesreceiving, via the machine and responsive to the custom survey data,survey results data.

Step 646 includes generating, utilizing machine learning and via themachine, a customized virtue scoring model based on the custom surveydata and the survey results data. Step 648 includes receiving, via themachine, content data. Step 650 includes generating, via the machine andutilizing the customized virtue scoring model, predicted virtue scoredata associated with the content data. Step 652 includes facilitatingdisplay, via the graphical user interface, the predicted virtue scoredata associated with the content data.

In addition or in the alternative, the customized virtue scoring modelincludes an artificial intelligence (AI) model that is trained based onat least one of: the custom survey data or the survey results data.

In addition or in the alternative, the AI model includes aresponsibility model and the predicted virtue score data indicates aresponsibility score that is based on an amount the content dataaddresses legal or ethical principles.

In addition or in the alternative, the AI model includes an equitabilitymodel and the predicted virtue score data indicates an equitabilityscore that is based on an amount of bias in the content data.

In addition or in the alternative, the AI model includes a reliabilitymodel and the predicted virtue score data indicates a reliability scorethat indicates variations in an others virtue scores.

In addition or in the alternative, the AI model includes anexplainability model and the predicted virtue score data indicates anexplainability score associated with the content data.

In addition or in the alternative, the AI model includes a moralitymodel and the predicted virtue score data indicates a morality scoreassociated with the content data.

In addition or in the alternative, the method further includesgenerating improvement data associated with and the predicted virtuescore data.

In addition or in the alternative, the content data is an ArtificialIntelligence (AI) model.

In addition or in the alternative, the method further includesfacilitating selection of the content data from at least one of: an AImodel, or a content source.

In addition or in the alternative, the customized virtue scoring modelincludes an artificial intelligence (AI) model and wherein generatingthe customized virtue scoring model includes providing access to aplurality of AI analysis widgets to facilitate an evaluation of the AImodel.

In addition or in the alternative, the plurality of AI analysis widgetsinclude a plurality of virtue scoring models that predict virtue scoresfor the AI model associated with a plurality of virtues.

In addition or in the alternative, the customized virtue scoring modelincludes an artificial intelligence (AI) model and wherein the methodfurther comprises providing access to a version control repository forstoring a plurality of versions of a training dataset and a plurality ofversion of the AI model.

FIG. 2D presents a flowchart representation of an example method inaccordance with various embodiments. In particular, a method 660 ispresented for use with any of the functions and features discussed inconjunction with FIGS. 1A-1E and/or the methods of FIGS. 2A, 2B and/or2C . Furthermore, a system comprising a network interface configured tocommunicate via a network, at least one processor and a non-transitorymachine-readable storage medium can store operational instructions that,when executed by the at least one processor, cause the at least oneprocessor to perform operations that any of the method steps describedbelow.

Step 662 includes generating, via a machine that includes at least oneprocessor and a non-transitory machine-readable storage medium andutilizing a graphical user interface, a content analysis control panel.Step 664 includes receiving, via the machine, content data.

Step 666 includes detecting, via one or more AI models implemented viathe machine, detection data that includes first portions of the contentdata associated with a protected attribute and second portions of thecontent data associated with a predetermined metric. Step 668 includesgenerating, via the machine, analysis data associated with the protectedattribute and the predetermined metric. Step 670 includes facilitatingdisplay, via the content analysis control panel, the analysis dataassociated with the protected attribute and the predetermined metric.

In addition or in the alternative, the protected attribute is apotential source of discrimination.

In addition or in the alternative, the potential source ofdiscrimination is at least one of: gender, race, age, religion,ethnicity, sexual preference, or disability.

In addition or in the alternative, the predetermined metric is a keyperformance indicator that varies based on the potential source ofdiscrimination.

In addition or in the alternative, the predetermined metric is a termthat varies based on the potential source of discrimination.

In addition or in the alternative, the predetermined metric indicates atleast one grade point average.

In addition or in the alternative, the predetermined metric indicates atleast one salary.

In addition or in the alternative, the predetermined metric indicates atleast one job offers.

In addition or in the alternative, the predetermined metric indicates atleast one loan approval or disapproval.

In addition or in the alternative, the predetermined metric indicates atleast one credit score.

In addition or in the alternative, the predetermined metric indicates atleast one job promotion.

In addition or in the alternative, the predetermined metric indicates atleast one arrest.

FIG. 3A presents a block diagram representation of an example AIauto-detection model. In particular, an AI auto-detection model 302 isshown that is an example of an AI analysis tool/widget 808 and/or acontent analysis tool/widget 808′. AI auto-detection model 302 istrained via training data 306 to recognize portions of input data 300that contain or are predicted to contain, one or more protectedattributes or other metrics. In various examples, the input data 300 canbe AI input/output data of an underlying AI process to be analyzedand/or content data from other media content from a media source to beanalyzed.

In various examples, the protected attributes can include terms relatedto gender, sex, race, age, religion, ethnicity, sexual preference,disabilities or other terms associated with potential sources ofdiscrimination. The metrics can, for example, include one or more terms,key performance indicators (KPIs) or other factors that could be presentin the input data 300 and be vary based on such sources ofdiscrimination. Examples of such metrics include grade point average,salary, job offers, loan approvals or disapprovals, credit scores,promotions, arrests, etc. depending on the type of data being analyzed.

In various examples, the AI auto-detection model 302 uses deep layerednatural language processing or other AI that is trained based ontraining data 306 that contains these terms, region variations, commonor expected misspellings of these terms, alternative terms, etc. Inoperation, the AI auto-detection model 302 generates detection data 304,such as columnar or tabular data containing labels that indicate theterms identified in the input data 300. While the AI auto-detectionmodel 302 is shown as a single model, the AI auto-detection model 302may contain a plurality of individual AI models, for example, eachtrained to recognize one corresponding term to be detected.

FIG. 3B presents a block diagram representation of an exampleauto-mapping function. In particular, an auto-mapping function 312 isshown that is a further example of an AI analysis tool/widget 808 and/ora content analysis tool/widget 808′.

In various embodiments, the auto-mapping function 312 operates on thedetection data 304 and applies a continuous distribution, categoricaldistribution, binned distribution or other statistical analysis togenerate analysis data indicating statistics and/or other valuesregarding protected attributes and metrics. Illustrative examples,include:

-   race = 36% white/Asian, 64% other races-   Age = 23% over 65-   GPA = 3.213 +/- 0.53-   LSAT score = 32 +/- 7-   Etc.

The auto-mapping function 312 can be implemented via one or moreparametric or non-parametric statistical functions. In other examples,the auto-mapping function 312 can be implemented via AI techniques andoptionally be trained based on training data 316 to generate theanalysis data 314. While the auto-mapping function 312 is shown as asingle function, the auto-mapping function 312 may contain a pluralityof individual functions, for example, each operable to generatestatistics or other analysis data 314 for a corresponding term or set ofterms indicated by the detection data 304.

FIG. 3C presents a block diagram representation of an example virtuescoring model. In particular, a virtue scoring model 322 is shown thatis a further example of an AI analysis tool/widget 808 and/or a contentanalysis tool/widget 808′. In operation, the virtue scoring model 322 istrained, for example, via training data 326 to generate a virtue score324 corresponding to a particular virtue in response to content data 320such as analysis data 314, AI output data of an underlying AI process tobe analyzed and/or content data from other media content from a mediasource to be analyzed. While the virtue scoring model 322 is shown as asingle model, the virtue scoring model 322 may contain a plurality ofindividual models, each corresponding to a different standard orcustomized virtue score 324.

Examples of the virtue scoring model(s) 322 include:

-   A responsibility scoring model trained to generate a virtue score    324 indicating a responsibility score or other metric that    indicates, for example, how well underlying AI or other content is    addressing legal and/or ethical principles;-   An equitability scoring model trained to generate a virtue score 324    indicating an equitability score or other metric, that indicates,    for example, an amount (or lack of) bias in the underlying AI or    other content data;-   A reliability scoring model or other function that generates a    virtue score 324 indicating that identifies variations or drift in    other virtue scores 324 or other changes in AI input or output data    from the training set that can. For example, indicate the need to    retrain the underlying AI or investigate the cause of changes in    scores in content data;-   An explainability scoring model trained to generate a virtue score    324 indicating an explainability score or other metric indicating,    for example, how transparent an underlying AI process is;-   One or more sub-models relating to portions of the results above,    that for example, can be used to construct overall virtue scores    324; and-   One or more user customized virtues, trained for example, based on    results from user defined surveys to generate other virtue scores    324 that are different than those listed above and address a    particular user problem or concern.

FIG. 3D presents a block/flow diagram representation of an examplesurvey creation process. As previously discussed, the AI developmentplatform 800 and content analysis platform 875 are operable to generatecustomized virtue scoring models that are trained or otherwise generatedbased on custom survey data and the survey results data. In the example,a survey creation widget 342 that is a further example of an AI analysistool/widget 808 and/or a content analysis tool/widget 808′ is used tocreate a custom survey 344 based on custom survey data 340 input by theuser via, for example, the content analysis control panel. The surveyresults data 348 are generated based on survey input 346 from surveyparticipants.

While the custom survey 344 is shown as a single survey, the surveycreation widget 342 can be used to generate multiple custom surveys formultiple custom virtue scoring models. Furthermore, survey data results348 and custom survey data 340 generated in this fashion can also beused to train any of the standard virtue scoring models discussed above.

FIG. 3E presents a pictorial/block diagram representation of an exampleof control panel generation tools 806. In the example shown controlpanel generation tools store control panel setting and customizationparameters 352 that are generated via interaction the user and userinput 350. In operation, the control panel generation tools 806 generatecontent analysis control panel data 354, based on further user input 350and the AI analysis tools/widgets 808 or content analysis tools/widgets808′. This content analysis control panel data 354 is formatted fordisplay via a display device of a client device, such as client device825 to reproduce the content analysis control panel 360. An examplescreen display is shown in FIG. 3F.

As previously discussed, the content analysis control panel 360generated via the set of control panel generation tools 806 operates inconjunction with the AI analysis tools/widgets 808 to provide agraphical user interface that aids the user by gathering and presentingAI data and/or other content for analysis, the creation of custom virtuescoring models, the selection of particular virtue scoring models(either custom or preset) to be used, and the presentation of virtuescores and other analysis results. For example, the content analysiscontrol panel 360 operates via the control panel generation tools 806and associated AI analysis tools/widgets 808 to:

-   guide the user through customization of control panel settings and    customization parameters 352 used to generate the content analysis    control panel 360;-   facilitate the selection of data sets from an AI model or content    source in addition to the selection of protected attributes, key    performance indicators and/or other metrics;-   identify, map and present data associated with the protected    attributes, key performance indicators and/or other metrics    including a customized selection of statistics, charts, graphs    and/or other visualizations;-   facilitate the generation of survey data and collection of survey    results data to facilitate the generation of custom and/or standard    virtue scoring models;-   generate and present virtue scores associated with a selected group    of virtue scoring models including a customized selection of    statistics, charts, graphs and/or other visualizations of each    score; and-   generate and present suggested improvements to any of the virtue    scores associated with a selected group of virtue scoring models.

FIGS. 4A - 4V and 5A - 5E present graphical diagram representations ofexample screen displays or portions thereof corresponding to a contentanalysis control panel. In particular, FIG. 4A presents a screen displayof a content analysis control panel (CACP) of a User “Jane Doe”. TheCACP includes a news feed that shows various AI related articles thatcan be individually accessed and read by the user. In FIG. 4B, the userhas accessed a drop-down menu and chosen to create a new AI pipeline. InFIG. 4C, a popup window is shown that allows the user to input a titleand description of the new pipeline. In, FIG. 4D, the CACP is shownafter the user has chosen to name the new pipeline, “medical treatmentselection pipeline”. The screen display indicates that there iscurrently no data for the pipeline and prompts to user to import data inorder to get started. In particular, the user has the option of dragginga dropping a data set into the window or using an API of the system.

In FIG. 4E, the user has imported a dataset and auto-detection andauto-mapping have been performed by the AI analysis widgets/tools 808 tocategorize the metrics “last”, “ugpa” and “zfgpa” by race, either“white/Asian” or “other”. Input data sets can, for example, be incolumnar format with columns representing different datatypes. Inputdata sets can be static, continuous updated and/or updated, periodically(e.g. once a day, once a week, once a month, etc.). In FIG. 4F, the userhas elected to view a history of datasets that have been entered, theirrespective dates and who they were added by (“Rory”, in this case).

In FIG. 4G, the user has customized the CACP by entering customizationdata to select and generate two particular virtue scoring models for theselected pipeline, a responsible/responsibility scoring model and aequitable/equitability scoring model.

Furthermore, the user has selected presentation parameters, eitherdefault or customized for each scoring model to indicate how the virtuescores will be displayed, for example, by particular graphs, charts, orother graphics or visual indications. In this case, the CACP prompts theuser to fill out a survey in order to train the responsibility scoringmodel. Equitability scores are presented in a window below in the chosenpresentation format along with an overall fairness index in the upperright portion of the screen. This fairness index can be generated basedon a function/combination of the user-selected virtues or based on allvirtues, depending on the implementation.

As shown in the panel on the right, robustness and traceability scoringmodels are also available as well as links to tools that assist the userin improving, responsibility, equitability, robustness and/or otherstandard virtues. As shown in the panel on the left, the user is giventhe options to retrain or deploy any of the selected virtue scoringmodels. Icons can also be provided allowing the user to seek human inthe loop (HIL) feedback and/or to share results with private groups,public groups, social media, etc.

In FIG. 4H, the equitability scoring window/bias monitor is selected andseveral different data overviews are presented in various and possiblyuser selected formats. In FIGS. 4I and 4J, the explainability scoringwindow/bias monitor is selected and several different data overviews arepresented in various and possibly user selected formats. In FIG. 4I, amacro-view of a data overview is shown that breaks down a “loan” metriconto four different components. In FIG. 4J, a micro-view is shown wheretotal/overall score (“good”) is presented along with a breakdown ofvarious inputs/features that contribute to that score. A prompt isprovided that allows the user to retrain the explainer (e.g., theexplainability virtue scoring model). In the panel on the right, theuser can query the system on the effects of selected features and how tochange certain features to receive certain scores, for example. Inaddition, the user is presented an option to create an extension.

FIG. 4K presents an interface on the CACP that uses the control panelgeneration tools to permit the user to create one or more customizedcontrol panels. Templates are available related to the categories“healthcare” and “finance” for users that want to start from apre-existing control panel configuration, as well as a blank templatefor users that wish to start from scratch. As indicated, control panelscan be designated as either public or private. As shown in the bottom ofthe screen display, a user that does not see a feature he/she wants,they may add feedback to the administrator of the platform to perhapsinclude this in a later release.

In FIG. 4L, the user has selected to create a new control panel andprompted to enter a control panel name. The user is also allowed tocreate a scoring model for a new/customized virtue. In FIG. 4M, the userhas used a survey widget to create a survey for a new virtue “Virtue I”.FIG. 4N, an example of the survey widget are shown. In FIG. 4O, the userselects the audience for completing the survey based on particular namesand email addresses - i.e. to generate survey input/results. The usercan select an existing crowd, employees for example, create a new crowdas shown in FIG. 4P, or proceed on a general crowd source. A screendisplay generated by the survey widget for a new survey is presented inFIG. 4Q.

FIG. 4R presents a cloud portal of the CACP that presents variousservice guides and a link to the news feed of FIG. 4A. In FIG. 4S, theuser is selecting to access the API reference materials. FIG. 4Tpresents a static/predetermined survey, magnitude slider that can beused to customize an AI analysis widget corresponding to the biasmonitor and equitability scoring model to enable virtue tracing based onscoring magnitudes. FIG. 4U presents a widget creator that allows a userto create/customize his/her own widgets. In FIGS. 4V - 4X, the user hasselected different output formats for display in conjunction with the AIanalysis widget corresponding to the bias monitor and equitabilityscoring model. FIG. 4Y presents a billing and payment screen.

In FIG. 5A, the survey widget configures a survey for multiple-choicequestions. In FIG. 5B, the survey widget configures a survey withmultiple-choice questions with answers input by users via slider-bars.In FIG. 5C, the survey widget configures a survey with short answersinput by users. In FIG. 5D, API options and instructions are provided tofacilitate the input of datasets.

FIGS. 6A - 6F present graphical diagram representations of examplescreen displays or portions thereof of another example content analysiscontrol panel. In particular, example screen displays are presented aspart of the graphical user interface implemented via the AI developmentplatform 800.

In various embodiments, the AI development platform 800 supports acommunal development framework that allows users to view repositories onpeople’s walls, view other profiles to see public work, promote trustthrough transparency, allow people to be involved in decisions, addfriends and follow people and organizational work, approve/disapprovework, borrow others code by forking or cloning their repository. Thiscommunal development framework also supports AI ethics discussion inethics forums, and/or other forums where a user posts a question, otherscan answer, and users can comment on question and answers. Documentationcan be provided in a “Learn” section which includes information on AIhow to use Version Control, Data API, an AI moral insight model, etc. Invarious embodiments, only users/subscribers are allowed to post, butothers can look at questions and answers.

In various embodiments, this communal development framework alsosupports a news feed that allows users to educate themselves on machinelearning, ethics, current events in AI ethics, etc. Users can alsocreate their own content. Tools can be provided to aid users in settingthe tone of their contributions and otherwise to provide a guide on howto post. This communal development framework also supportsorganizational billing for cloud services allowing users to, forexample, choose their organization with billing credentials and printout a quick report. Variable subscription plans can be offered thatallow users to subscribe to the specific services and/or level of usethey may need.

As used herein the terms “widget”, “tool” and “toolkit” correspond to awebsite, utility, platform, computer, cloud device and/or softwareroutine that performs one or more specific functions.

It is noted that terminologies as may be used herein such as bit stream,stream, signal sequence, etc. (or their equivalents) have been usedinterchangeably to describe digital information whose contentcorresponds to any of a number of desired types (e.g., data, video,speech, text, graphics, audio, etc. any of which may generally bereferred to as ‘data’).

As may be used herein, the terms “substantially” and “approximately”provide an industry-accepted tolerance for its corresponding term and/orrelativity between items. For some industries, an industry-acceptedtolerance is less than one percent and, for other industries, theindustry-accepted tolerance is 10 percent or more. Other examples ofindustry-accepted tolerance range from less than one percent to fiftypercent. Industry-accepted tolerances correspond to, but are not limitedto, component values, integrated circuit process variations, temperaturevariations, rise and fall times, thermal noise, dimensions, signalingerrors, dropped packets, temperatures, pressures, material compositions,and/or performance metrics. Within an industry, tolerance variances ofaccepted tolerances may be more or less than a percentage level (e.g.,dimension tolerance of less than +/- 1%). Some relativity between itemsmay range from a difference of less than a percentage level to a fewpercent. Other relativity between items may range from a difference of afew percent to magnitude of differences.

As may also be used herein, the term(s) “configured to”, “operablycoupled to”, “coupled to”, and/or “coupling” includes direct couplingbetween items and/or indirect coupling between items via an interveningitem (e.g., an item includes, but is not limited to, a component, anelement, a circuit, and/or a module) where, for an example of indirectcoupling, the intervening item does not modify the information of asignal but may adjust its current level, voltage level, and/or powerlevel. As may further be used herein, inferred coupling (i.e., where oneelement is coupled to another element by inference) includes direct andindirect coupling between two items in the same manner as “coupled to”.

As may even further be used herein, the term “configured to”, “operableto”, “coupled to”, or “operably coupled to” indicates that an itemincludes one or more of power connections, input(s), output(s), etc., toperform, when activated, one or more its corresponding functions and mayfurther include inferred coupling to one or more other items. As maystill further be used herein, the term “associated with”, includesdirect and/or indirect coupling of separate items and/or one item beingembedded within another item.

As may be used herein, the term “compares favorably”, indicates that acomparison between two or more items, signals, etc., indicates anadvantageous relationship that would be evident to one skilled in theart in light of the present disclosure, and based, for example, on thenature of the signals/items that are being compared. As may be usedherein, the term “compares unfavorably”, indicates that a comparisonbetween two or more items, signals, etc., fails to provide such anadvantageous relationship and/or that provides a disadvantageousrelationship. Such an item/signal can correspond to one or more numericvalues, one or more measurements, one or more counts and/or proportions,one or more types of data, and/or other information with attributes thatcan be compared to a threshold, to each other and/or to attributes ofother information to determine whether a favorable or unfavorablecomparison exists. Examples of such a advantageous relationship caninclude: one item/signal being greater than (or greater than or equalto) a threshold value, one item/signal being less than (or less than orequal to) a threshold value, one item/signal being greater than (orgreater than or equal to) another item/signal, one item/signal beingless than (or less than or equal to) another item/signal, oneitem/signal matching another item/signal, one item/signal substantiallymatching another item/signal within a predefined or industry acceptedtolerance such as 1%, 5%, 10% or some other margin, etc. Furthermore,one skilled in the art will recognize that such a comparison between twoitems/signals can be performed in different ways. For example, when theadvantageous relationship is that signal 1 has a greater magnitude thansignal 2, a favorable comparison may be achieved when the magnitude ofsignal 1 is greater than that of signal 2 or when the magnitude ofsignal 2 is less than that of signal 1. Similarly, one skilled in theart will recognize that the comparison of the inverse or opposite ofitems/signals and/or other forms of mathematical or logical equivalencecan likewise be used in an equivalent fashion. For example, thecomparison to determine if a signal X > 5 is equivalent to determiningif -X< -5, and the comparison to determine if signal A matches signal Bcan likewise be performed by determining -A matches -B or not(A) matchesnot(B). As may be discussed herein, the determination that a particularrelationship is present (either favorable or unfavorable) can beutilized to automatically trigger a particular action. Unless expresslystated to the contrary, the absence of that particular condition may beassumed to imply that the particular action will not automatically betriggered. In other examples, the determination that a particularrelationship is present (either favorable or unfavorable) can beutilized as a basis or consideration to determine whether to perform oneor more actions. Note that such a basis or consideration can beconsidered alone or in combination with one or more other bases orconsiderations to determine whether to perform the one or more actions.In one example where multiple bases or considerations are used todetermine whether to perform one or more actions, the respective basesor considerations are given equal weight in such determination. Inanother example where multiple bases or considerations are used todetermine whether to perform one or more actions, the respective basesor considerations are given unequal weight in such determination.

As may be used herein, one or more claims may include, in a specificform of this generic form, the phrase “at least one of a, b, and c” orof this generic form “at least one of a, b, or c”, with more or lesselements than “a”, “b”, and “c”. In either phrasing, the phrases are tobe interpreted identically. In particular, “at least one of a, b, and c”is equivalent to “at least one of a, b, or c” and shall mean a, b,and/or c. As an example, it means: “a” only, “b” only, “c” only, “a” and“b”, “a” and “c”, “b” and “c”, and/or “a”, “b”, and “c”.

As may also be used herein, the terms “processing module”, “processingcircuit”, “processor”, “processing circuitry”, and/or “processing unit”may be a single processing device or a plurality of processing devices.Such a processing device may be a microprocessor, microcontroller,digital signal processor, microcomputer, central processing unit, fieldprogrammable gate array, programmable logic device, state machine, logiccircuitry, analog circuitry, digital circuitry, and/or any device thatmanipulates signals (analog and/or digital) based on hard coding of thecircuitry and/or operational instructions. The processing module,module, processing circuit, processing circuitry, and/or processing unitmay be, or further include, memory and/or an integrated memory element,which may be a single memory device, a plurality of memory devices,and/or embedded circuitry of another processing module, module,processing circuit, processing circuitry, and/or processing unit. Such amemory device may be a read-only memory, random access memory, volatilememory, non-volatile memory, static memory, dynamic memory, flashmemory, cache memory, and/or any device that stores digital information.Note that if the processing module, module, processing circuit,processing circuitry, and/or processing unit includes more than oneprocessing device, the processing devices may be centrally located(e.g., directly coupled together via a wired and/or wireless busstructure) or may be distributedly located (e.g., cloud computing viaindirect coupling via a local area network and/or a wide area network).Further note that if the processing module, module, processing circuit,processing circuitry and/or processing unit implements one or more ofits functions via a state machine, analog circuitry, digital circuitry,and/or logic circuitry, the memory and/or memory element storing thecorresponding operational instructions may be embedded within, orexternal to, the circuitry comprising the state machine, analogcircuitry, digital circuitry, and/or logic circuitry. Still further notethat, the memory element may store, and the processing module, module,processing circuit, processing circuitry and/or processing unitexecutes, hard coded and/or operational instructions corresponding to atleast some of the steps and/or functions illustrated in one or more ofthe Figures. Such a memory device or memory element can be included inan article of manufacture.

One or more embodiments have been described above with the aid of methodsteps illustrating the performance of specified functions andrelationships thereof. The boundaries and sequence of these functionalbuilding blocks and method steps have been arbitrarily defined hereinfor convenience of description. Alternate boundaries and sequences canbe defined so long as the specified functions and relationships areappropriately performed. Any such alternate boundaries or sequences arethus within the scope and spirit of the claims. Further, the boundariesof these functional building blocks have been arbitrarily defined forconvenience of description. Alternate boundaries could be defined aslong as the certain significant functions are appropriately performed.Similarly, flow diagram blocks may also have been arbitrarily definedherein to illustrate certain significant functionality.

To the extent used, the flow diagram block boundaries and sequence couldhave been defined otherwise and still perform the certain significantfunctionality. Such alternate definitions of both functional buildingblocks and flow diagram blocks and sequences are thus within the scopeand spirit of the claims. One of average skill in the art will alsorecognize that the functional building blocks, and other illustrativeblocks, modules and components herein, can be implemented as illustratedor by discrete components, application specific integrated circuits,processors executing appropriate software and the like or anycombination thereof.

In addition, a flow diagram may include a “start” and/or “continue”indication. The “start” and “continue” indications reflect that thesteps presented can optionally be incorporated in or otherwise used inconjunction with one or more other routines. In addition, a flow diagrammay include an “end” and/or “continue” indication. The “end” and/or“continue” indications reflect that the steps presented can end asdescribed and shown or optionally be incorporated in or otherwise usedin conjunction with one or more other routines. In this context, “start”indicates the beginning of the first step presented and may be precededby other activities not specifically shown. Further, the “continue”indication reflects that the steps presented may be performed multipletimes and/or may be succeeded by other activities not specificallyshown. Further, while a flow diagram indicates a particular ordering ofsteps, other orderings are likewise possible provided that theprinciples of causality are maintained.

The one or more embodiments are used herein to illustrate one or moreaspects, one or more features, one or more concepts, and/or one or moreexamples. A physical embodiment of an apparatus, an article ofmanufacture, a machine, and/or of a process may include one or more ofthe aspects, features, concepts, examples, etc. described with referenceto one or more of the embodiments discussed herein. Further, from figureto figure, the embodiments may incorporate the same or similarly namedfunctions, steps, modules, etc. that may use the same or differentreference numbers and, as such, the functions, steps, modules, etc. maybe the same or similar functions, steps, modules, etc. or differentones.

Unless specifically stated to the contra, signals to, from, and/orbetween elements in a figure of any of the figures presented herein maybe analog or digital, continuous time or discrete time, and single-endedor differential. For instance, if a signal path is shown as asingle-ended path, it also represents a differential signal path.Similarly, if a signal path is shown as a differential path, it alsorepresents a single-ended signal path. While one or more particulararchitectures are described herein, other architectures can likewise beimplemented that use one or more data buses not expressly shown, directconnectivity between elements, and/or indirect coupling between otherelements as recognized by one of average skill in the art.

The term “module” is used in the description of one or more of theembodiments. A module implements one or more functions via a device suchas a processor or other processing device or other hardware that mayinclude or operate in association with a memory that stores operationalinstructions. A module may operate independently and/or in conjunctionwith software and/or firmware. As also used herein, a module may containone or more sub-modules, each of which may be one or more modules.

As may further be used herein, a computer readable memory includes oneor more memory elements. A memory element may be a separate memorydevice, multiple memory devices, or a set of memory locations within amemory device. Such a memory device may be a read-only memory, randomaccess memory, volatile memory, non-volatile memory, static memory,dynamic memory, flash memory, cache memory, a quantum register or otherquantum memory and/or any other device that stores data in anon-transitory manner. Furthermore, the memory device may be in a formof a solid-state memory, a hard drive memory or other disk storage,cloud memory, thumb drive, server memory, computing device memory,and/or other non-transitory medium for storing data. The storage of dataincludes temporary storage (i.e., data is lost when power is removedfrom the memory element) and/or persistent storage (i.e., data isretained when power is removed from the memory element). As used herein,a transitory medium shall mean one or more of: (a) a wired or wirelessmedium for the transportation of data as a signal from one computingdevice to another computing device for temporary storage or persistentstorage; (b) a wired or wireless medium for the transportation of dataas a signal within a computing device from one element of the computingdevice to another element of the computing device for temporary storageor persistent storage; (c) a wired or wireless medium for thetransportation of data as a signal from one computing device to anothercomputing device for processing the data by the other computing device;and (d) a wired or wireless medium for the transportation of data as asignal within a computing device from one element of the computingdevice to another element of the computing device for processing thedata by the other element of the computing device. As may be usedherein, a non-transitory computer readable memory is substantiallyequivalent to a computer readable memory. A non-transitory computerreadable memory can also be referred to as a non-transitory computerreadable storage medium.

One or more functions associated with the methods and/or processesdescribed herein can be implemented via a processing module thatoperates via the non-human “artificial” intelligence (AI) of a machine.Examples of such AI include machines that operate via anomaly detectiontechniques, decision trees, association rules, expert systems and otherknowledge-based systems, computer vision models, artificial neuralnetworks, convolutional neural networks, support vector machines (SVMs),Bayesian networks, genetic algorithms, feature learning, sparsedictionary learning, preference learning, deep learning and othermachine learning techniques that are trained using training data viaunsupervised, semi-supervised, supervised and/or reinforcement learning,and/or other AI. The human mind is not equipped to perform such AItechniques, not only due to the complexity of these techniques, but alsodue to the fact that artificial intelligence, by its very definition —requires “artificial” intelligence — i.e. machine/non-humanintelligence.

One or more functions associated with the methods and/or processesdescribed herein can be implemented as a large-scale system that isoperable to receive, transmit and/or process data on a large-scale. Asused herein, a large-scale refers to a large number of data, such as oneor more kilobytes, megabytes, gigabytes, terabytes or more of data thatare received, transmitted and/or processed. Such receiving, transmittingand/or processing of data cannot practically be performed by the humanmind on a large-scale within a reasonable period of time, such as withina second, a millisecond, microsecond, a real-time basis or other highspeed required by the machines that generate the data, receive the data,convey the data, store the data and/or use the data.

One or more functions associated with the methods and/or processesdescribed herein can require data to be manipulated in different wayswithin overlapping time spans. The human mind is not equipped to performsuch different data manipulations independently, contemporaneously, inparallel, and/or on a coordinated basis within a reasonable period oftime, such as within a second, a millisecond, microsecond, a real-timebasis or other high speed required by the machines that generate thedata, receive the data, convey the data, store the data and/or use thedata.

One or more functions associated with the methods and/or processesdescribed herein can be implemented in a system that is operable toelectronically receive digital data via a wired or wirelesscommunication network and/or to electronically transmit digital data viaa wired or wireless communication network. Such receiving andtransmitting cannot practically be performed by the human mind becausethe human mind is not equipped to electronically transmit or receivedigital data, let alone to transmit and receive digital data via a wiredor wireless communication network.

One or more functions associated with the methods and/or processesdescribed herein can be implemented in a system that is operable toelectronically store digital data in a memory device. Such storagecannot practically be performed by the human mind because the human mindis not equipped to electronically store digital data.

One or more functions associated with the methods and/or processesdescribed herein may operate to cause an action by a processing moduledirectly in response to a triggering eventwithout any intervening humaninteraction between the triggering event and the action. Any suchactions may be identified as being performed “automatically”,“automatically based on” and/or “automatically in response to” such atriggering event. Furthermore, any such actions identified in such afashion specifically preclude the operation of human activity withrespect to these actions - even if the triggering event itself may becausally connected to a human activity of some kind.

While particular combinations of various functions and features of theone or more embodiments have been expressly described herein, othercombinations of these features and functions are likewise possible. Thepresent disclosure is not limited by the particular examples disclosedherein and expressly incorporates these other combinations.

What is claimed is:
 1. A method comprising: generating, via a machinethat includes at least one processor and a non-transitorymachine-readable storage medium and utilizing a graphical userinterface, a content analysis control panel; receiving, via the machine,content data; detecting, via one or more AI models implemented via themachine, detection data that includes first portions of the content dataassociated with a protected attribute and second portions of the contentdata associated with a predetermined metric; generating, via themachine, analysis data associated with the protected attribute and thepredetermined metric; and facilitating display, via the content analysiscontrol panel, the analysis data associated with the protected attributeand the predetermined metric.
 2. The method of claim 1, wherein theprotected attribute is a potential source of discrimination.
 3. Themethod of claim 2, wherein the potential source of discrimination is atleast one of: gender, race, age, religion, ethnicity, sexual preference,or disability.
 4. The method of claim 3, wherein the predeterminedmetric is a key performance indicator that varies based on the potentialsource of discrimination.
 5. The method of claim 3, wherein thepredetermined metric is a term that varies based on the potential sourceof discrimination.
 6. The method of claim 3, wherein the predeterminedmetric indicates at least one grade point average.
 7. The method ofclaim 3, wherein the predetermined metric indicates at least one salary.8. The method of claim 3, wherein the predetermined metric indicates atleast one job offers.
 9. The method of claim 3, wherein thepredetermined metric indicates at least one loan approval ordisapproval.
 10. The method of claim 3, wherein the predetermined metricindicates at least one credit score.
 11. The method of claim 3, whereinthe predetermined metric indicates at least one job promotion.
 12. Themethod of claim 3, wherein the predetermined metric indicates at leastone arrest.
 13. A system comprises: a network interface configured tocommunicate via a network; at least one processor; a non-transitorymachine-readable storage medium that stores operational instructionsthat, when executed by the at least one processor, cause the at leastone processor to perform operations that include: generating, via the atleast one processor and utilizing a graphical user interface, a contentanalysis control panel; receiving, via the at least one processor,content data; detecting, via one or more AI models implemented via theat least one processor, detection data that includes first portions ofthe content data associated with a protected attribute and secondportions of the content data associated with a predetermined metric;generating, via the at least one processor, analysis data associatedwith the protected attribute and the predetermined metric; andfacilitating display, via the content analysis control panel, theanalysis data associated with the protected attribute and thepredetermined metric.
 14. The system of claim 13, wherein the protectedattribute is a potential source of discrimination, and wherein thepotential source of discrimination is at least one of: gender, race,age, religion, ethnicity, sexual preference, or disability.
 15. Thesystem of claim 14, wherein the predetermined metric indicates at leastone grade point average.
 16. The system of claim 14, wherein thepredetermined metric indicates at least one salary.
 17. The system ofclaim 14, wherein the predetermined metric indicates at least one joboffers.
 18. The system of claim 14, wherein the predetermined metricindicates at least one loan approval or disapproval.
 19. The system ofclaim 14, wherein the predetermined metric indicates at least one creditscore.
 20. The system of claim 14, wherein the predetermined metricindicates at least one job promotion or at least one arrest.